The invention described in the aforesaid application answers a need, a requirement to image and digitally record an object in a relatively flat plane at high resolution/magnification. Today, it is impractical to construct an optical image sensor large enough to cover the entire image area e.g., of a specimen on a microscope slide, at the required resolution. This is because lens size and resolution/magnification issues limit the size of the field of view of magnified objects and their resulting images. Viewing through a microscope is akin to viewing through a periscope in that one sees a very small field of view even at low magnifications, such as 1.25.times.. A pathologist using a microscope often scans a slide to obtain in his mind an overall view or sense of what constitutes the specimen and he remembers the general locations of the diagnostically significant, small pieces of the specimen. Usually, these are the diseased areas, such as malignant or potentially malignant portions of the specimen. To obtain higher resolution and magnification of these suspicious portions, the pathologist switches to a higher magnification objective lens but then the field of view becomes much smaller again. Often, the pathologist switches back and forth between the lower magnification, larger field of view objective lens to orient himself relative to the specimen and the high magnification, smaller field of view to obtain the detailed, high resolution view of the suspicious area on the specimen. Thus, the user never receives a magnified, condensed overall view of the specimen or a portion of the specimen but must remember the series of views taken at low magnification. Likewise, at high resolution, high magnification, the user never receives or views a collection of adjacent images but must interrelate these successive images in the user's mind.
A similar problem exists on the Internet or intranet where a pathologist may receive a single field of view magnified image taken from a specimen over the Internet or the intranet on his browser. The pathologist must be provided with explanations to coordinate the high resolution view with the lower resolution view. The number of views available to the pathologist is very limited, and the pathologist is unable to select other views or to scroll to neighboring views at the areas that are most interesting to the pathologist.
In the aforesaid prior application, there is disclosed a method and apparatus whereby a person may construct a low magnification, digitized overall, image view of the entire specimen on a slide or a selected portion of the specimen on a slide, such as the basal layer of a tissue section. The overall, low magnification digitized image allows the user to understand where the user is presently located in his viewing and where the user may want to make the next observation. That is, the low magnification overall view is generally in color and provides to the experienced user a visual overall or thumbnail view of the slide and shows the possible areas of interest for malignancy or other diseases which manifest themselves at certain locations on the specimen image being viewed. This low magnification overall view enables the user to select thereon the points of interest that the user wants to view at a higher magnification.
The overall view was constructed by taking by a large number of low magnification images of the specimen through a microscopic scanning system and then coherently assembling and coordinating these respective smaller views or images (hereinafter "image tiles") into one coherent, low magnification, macro image from the specimen. Often the digitized macro image is reduced in size by a software system to even a smaller size to be displayed on a local screen or to be transferred over a low bandwidth or a high bandwidth channel to a remote viewing screen.
The prior application teaches how to assemble a large number of image tiles, for example, 35 image tiles for the macro image, and then to take a series of other tiles of a higher magnification or magnifications which will also be viewed by the user. To this end, the user is provided with a marker, such as a cursor or the like, to select the defined area of interest, and by a simple command, to cause the selected, higher magnification digitized images to appear on the screen for viewing by the user. The higher magnification images may be one of several magnifications or resolutions such as 10.times., 20.times. and 40.times..
As disclosed in the aforesaid application, it is preferred to allow the user, such as a pathologist, to quickly flip back and forth between the high resolution micro image and the low macro resolution image or to provide separate split screens whereby the pathologist is shown an overall macro view and a marker showing where the current higher magnification view is located. Because of the multiple magnifications, the user may change to an intermediate magnification such as would be accomplished by switching between intermediate objective lenses. This provides the pathologist with views which correspond to changing back and forth between objective lenses in a microscope, a procedure with which most pathologists are familiar and have been trained.
Additionally, the aforesaid application provides the user with a scrolling feature that allows the user to shift into the viewing screen adjacent, magnified images on the screen so that the pathologist is not limited to only seeing just a full tile view but may see adjacent image material from adjacent, neighboring tile images.
In the aforesaid patent application, there is a disclosure of transmitting the low magnification image over a local area network or over the Internet through various servers and computers. The tiled images that were being transmitted were achieved by use of a fully computer controlled microscope which allowed the user to navigate along a specimen area of interest, such as along a basal area or to other suspicious points spread throughout the specimen to acquire tiled images of selected areas so that the entire specimen would not have to be digitized and stored. As disclosed in the preferred embodiment in the aforesaid application, an Internet browser remotely-controlled, automated microscope could be used by a pathologist from a remote location to view the reconstructed macro image tiles; and, with his manipulation of the microscope, using an intranet or Internet browser, could acquire single images at higher magnifications if desired. While several people could see the particular digitized images being transmitted out over the Internet as they were acquired by a particular pathologist and several people could view the stored images, there was still a problem of control at operation of the microscope by each person viewing the digitized images, and a problem with acquiring and transmitting large areas of higher magnification images using the tiling method.
As stated above in greater detail, the current state of archiving the digital images achieved through a microscope is often by having photographs or by video tapes. The photographs are difficult to use as is a video tape particularly when the user wants to move rapidly back and forth between various images and to scroll through various adjacent parts of the specimen image. Further, current archival methods lack an overall macro image of the specimen, which allows the user to know exactly where the particular high resolution view is from when it is making an analysis of the high resolution image.
While digitized images can be stored magnetically or otherwise digitized and recorded on various recording mediums, no current archival system allows the user to toggle back and forth between high magnification images and low magnification images or between various images at different magnifications such as that achieved by a pathologist switching microscope objective lenses in real time to get the macro and micro images from the same location on the specimen. Heretofore, the practice of pathology has been relatively limited to the use of microscopes and to the pathologist having to use the microscope to review the particular specimen.
There is a need for a dynamic system whereby one or more or several pathologists, including a consulting pathologist, may view the same area simultaneously and interact with one another either in diagnosis or in analysis. Also, it would be best if the images from the specimen could be stored so that a pathologist could easily examine the images at his leisure using an intranet or Internet browser at a later date merely by accessing the particular web site where the images are located.
It will be appreciated that a host of problems need to be solved to allow Internet or intranet users to view on their respective monitors useful, low resolution, macro images and high resolution, micro images of several adjacent, original microscope images. One of the first problems is how to seam together neighboring tile images to form a seamless overall view of these tiles. Heretofore, attempts to seam the tiles used software to combine the pixels at the tile boundaries and have been generally unsuccessful. Another problem is that of mapping of coordinates beginning with the coordinates, usually X and Y coordinates, from and at the microscope stage carrying the slide and then the mapping of coordinates on the scanning screen not only for one magnification but also to coordinate the mapping for the respective multiple resolution images taken typically at 1.25.times., 10.times. and 40.times. or more. These coordinates must be maintained for a large number of tiled images, e.g., 40 tiled images for one macro image. In order for the remote user to view these tile images and to flip back and forth between different resolution, tiled images, the user's computer and monitor not only must receive the addresses and stored parameters for each pixel but must also run them on a generic viewing program.
Another problem with acquiring image tiles and sending them over a low bandwidth Internet channel is that both the storage requirements on the server and the amount of data acquired per slide become high, such as for example, 120 megabytes to one gigabyte. The 120 megabytes is only achieved by not taking image tiles of the entire specimen but only image tiles from the areas selected by the pathologist when tracing at high resolution along basal layers or only at the dispersed, suspicious cancer appearing area in a breast cancer. Even with this selective interaction by a pathologist in constructing the macro and micro digitized images with a vastly reduced amount of image tiles relative to that which would be acquired if the entire specimen where imaged at each of the multiple magnifications, the acquired amount of data is a monstrous problem of transmitting in a reasonable amount of time over a narrow bandwidth channel to an ordinary web browser having limited storage capacity. While rough compression techniques could be used, they cannot be used at the expense of providing the high resolution image that the pathologist must have for diagnosis of the specimen.